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Introduction 


In research, statistical significance is a measure of the probability of the null hypothesis being true compared to the acceptable 
level of uncertainty regarding the true answer. If we break apart a study design, we can better understand statistical significance.[1 ] 
[2] [3] [4] [5] [6] [7] 

When creating a study, the researcher has to start with a hypothesis, that is they must have some idea of what they think the 
outcome may be. We will use the example of a new medication to lower blood pressure. The researcher hypothesizes that the new 
medication lowers systolic blood pressure by at least 10 mmHg compared to not taking the new medication. The hypothesis can be 
then stated, "Taking the new medication will lower systolic blood pressure by at least 10 mmHg compared to not taking the 
medication." In science, researchers can never prove any statement as there are infinite alternatives as to why the outcome may 
have occurred. They can only try to disprove a specific hypothesis. The researcher must then formulate a question they can 
disprove while coming to their conclusion that the new medication lowers systolic blood pressure. The hypothesis, to be disproven, 
is the null hypothesis and typically the inverse statement of the hypothesis. Thus, the null hypothesis for our researcher would be 
"Taking the new medication will not lower systolic blood pressure by at least 10 mmHg compared to not taking the new 
medication." The researcher now has the null hypothesis for the research and must next specify the significance level or level of 
acceptable uncertainty. 


Even when disproving a hypothesis the researcher will not be 100% certain of the outcome. The researcher must then settle for 


some level of confidence or the significancejl^ai^a^i^/lpiQbl^lW jftft<j!grS9Gftcft ie significance level is given the Greek 


letter alpha and specified as the probability the researcherfi§|V\p|kjEigfpj£)|® incorrect. Our 


researcher wants to be correct about their 
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outcome 95% of the time, or the research e^i^llyilities are stated as decimals with 1.0 
being completely positive (100%) and 0 beit^g J 0(p 1 i^0L^^(-g | ^g^ / tji^49 l )^9^0^? t ^ e researcher who wants to be 95% sure about the 
outcome of their study is willing to be wronjg^/^ftfj-tte time about the study result. The alpha is the decimal expression of how 
much they are willing to be wrong. For the current example, the alpha is 0.05. We now have the level of uncertainty the researcher 
is willing to accept (alpha or significance level) of 0.05 or 5% chance they are not correct about the outcome of the study. 

Now the researcher can perform their research. In the example, the researcher would give some individuals the new medication 
and other individuals, no medication. The researcher then looks to see how the blood pressure changes after receiving the new 
medication and performs a statistical analysis of the results to obtain a p-value (probability value). There are numerous different 
tests used in research which can provide a p-value. It is imperative to use the correct statistical analysis tool when calculating the p- 
value. If the researchers use the wrong test, the p-value will not be accurate, and this result can mislead the researcher. The p-value 
is best described as the probability that the null hypothesis is true given the researcher's current set of data. In our example, the 
researcher found blood pressures did tend to decrease after taking the new medication. The researcher then used the help of 
their statistician to perform the correct analysis and arrives at a p-value of 0.02 for the decrease in blood pressure for those taking 
the new medication versus those not taking the new medication. Our researcher now has the three required pieces of information 
to look at statistical significance: the null hypothesis, the significance level, and the p-value. 

The researcher can finally assess the statistical significance regarding the new medication. A study result is stated to be statistically 
significant if the p-value of the data analysis is less than the prespecified alpha (significance level). In our example, the p-value is 
0.02 which is less than the pre-specified alpha of 0.05, so the researcher concludes there is statistical significance for the study. 

What does this mean? The p-value of 0.02 implies that there is a 2% chance of the null hypothesis being correct, true, or explained 
by the current set of data. Remember the null hypothesis states that there is no significant change in blood pressure if the patient 
is or is not taking the new medication. Thus in this example, there is only a 2% chance the null hypothesis is correct based on the 
obtained data. The researcher pre-specified an alpha of 0.05 implying they wanted the chance of the null hypothesis to be less 
than 5% before rejecting the null hypothesis. As the p-value is 0.02 and less than the alpha of 0.05 the researcher rejects the null 
hypothesis because there is statistical significance. By rejecting the null hypothesis, the researcher accepts the alternative 
hypothesis. The researcher rejects the idea that there is no difference in systolic blood pressure with the new medication and 
accepts the alternative that there is a difference of at least 10 mmHg in systolic blood pressure when taking the new medication 

If the researcher had prespecified an alpha of 0.01, implying they wanted to be 99% sure the new medication lowered the blood 
pressure by at least 10 mmHg, then the p-value of 0.02 would be greater than the prespecified alpha of 0.01. The researcher would 
conclude the study did not reach statistical significance as the p-value is equal to or greater than the pre-specified alpha. The 
research would then not be able to reject the null hypothesis. 


Function 
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A study is statistically significant if the p-val^^^ ^(^g^ fi ^gp^^g rW .ccinctly: 

. A p-value less than alpha is a statistic^^fr(M™ie{f d 14476968097 
• A p-value greater to or equal to alpha^s statistically significant result. 


{ssues of Concern 

There are a few issues of concern when looking at statistical significance. These issues include choosing the alpha, statistical 
analysis method, and clinical significance. 

Many current research articles specify an alpha of 0.05 for their significance level. It can not be stated strongly enough that there is 
nothing special, mathematical, or certain about picking an alpha of 0.05. Historically, the originators concluded that for many 
applications an alpha of 0.05 or a one in 20 chance of being incorrect was good enough. It is imperative for the researcher to 
consider what their confidence level should truly be for the research question they are asking. Many times a smaller alpha, say of 
0.01, may be more appropriate. 

When creating a study, the alpha, or confidence level, should be specified before any intervention or collection of data. It is easy for 
a researcher to "see what the data shows" then pick an alpha to give a statistically significant result. Such approaches compromise 
the data and results as the researcher is more likely to be lax on confidence level selection to obtain a result that looks statistically 
significant. 

A second important issue is selecting the correct statistical analysis method. There are numerous methods for obtaining a p-value. 
The method chosen depends on the type of data, number of data points, and the question being asked. It is important to consider 
these questions during the study design so the statistical analysis can be correctly identified before the research. The statistical 
analysis method can then help determine how to collect the data correctly as well as the number of data points needed. If the 
wrong statistical method is used, the results may be meaningless as an incorrect p-value would be calculated. 


Clinical Significance 

There is a key distinction in statistical significance versus clinical significance. Statistical significance determines if 
there is mathematical significance to the analysis of the results. Clinical significance means the difference is important to the 
patient and the clinician. In our study, the statistical significance would be present as the p-value was less than the pre-specified 
alpha. The clinical significance would be the 10 mmHg drop in systolic blood pressure.[6] 


Two studies can have a similar statistical si 
two new chemotherapy agents for treating 
and alpha for the study of 0.05. Thus, this 


nificance b ut vastly differ in dinical significance. Let us use a hypothetical example of 


g i>N>h/feiBcF(Rgrlf€l nsnioe cbl 


tudy has statistifaffigfiffitEifllfe (p-value less than alpha) and clinical significance 


by at least ten years with a p-value of 0.01 
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(increased survival by ten years). A second fail'd fbi the survival by at least 10 minutes 

with a p-value of 0.01 and alpha for the stu^g^Q.^g^^l^^fpi^g^g^^o found statistical significance (p-value less than 
alpha) but no clinical significance (a 10-min^te]ig^pg^ in life expectancy is not clinically significant). In a separate study, those 
taking Drug A lived on average eight years after starting the medication versus living for only two more years on average for those 
not taking Drug A with a p-value of 0.08 and alpha for this second study of Drug A of 0.05. In this second study of Drug A, there is 
no statistical significance (p-value greater to or equal alpha). 

Other Issues 

Even when a result is statistically significant, it may not be correct. A researcher may thoughtfully design a study with a confidence 
level of 95% (thus the alpha would be 0.05) and obtain a p-value of 0.04. As the p-value of 0.04 is less than the alpha, the study 
would be considered to have statistical significance. Based on the alpha of 0.05, the researcher is only 95% sure they are correct in 
their conclusion. There is thus a 5% chance that although the results are statistically significant the conclusion is nonetheless 
incorrect. 
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A study finds that there was statistical significance of the outcome measure. Select the correct statement. 


Answers Teaching Points 

Click Your Answer Below 

The findings of the study was most likely not due to chance 
The treatment studied was not effective 
The sample size was inadequate 

The treatment will make a significant difference in how patients are treated 


For a research project the p-value was found to be 0.03 with an alpha of 0.01. What is the best interpretation of the 
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Answers Teaching Points 
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Click Your Answer Below 

The project is statistically significant and the null hypothesis should be rejected 
The project is statistically significant and the null hypothesis should be accepted 
The project is not statistically significant and the null hypothesis should be rejected 
The project is not statistically significant and the null hypothesis can not be rejected 


A researcher is looking at prolonging survival for cancer patients after they receive a new cancer medication. The 
researcher prespecifies the significance level of 0.05. The analysis of the study finds a p-value of 0.03 for extending 
overall survival by 2 minutes. Which of the following statements is most correct? 


Answers Teaching Points 


Click Your Answer 
The study is 
The study is 
The study is 
The study is 


Below 


statistically and clinically significant 
statistically but not clinically significant 


not statistically significant but clinically significant 


neither statis tica ll y no r cl j ni cajjy signjficant 
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(https://itunes.apple.com/us/app/medicalpearls- 

A researcher is thinking about perfot|i!nb^^dsit£<iiyei1it^iic£ls^$$@3d^ows they must have both a p-value and 
significance level (alpha) to test for st&t1&ra&t3>)ignificance. When is the correct time during the research to set the 
significance level? 

Answers Teaching Points 

Click Your Answer Below 

Choose the alpha prior to starting the study 
Choose the alpha after the data is collected 
Choose the alpha after the p-value is calculated 
Calculate the alpha after the p-value is picked 


A researcher specifies their alpha as 0.05 prior to starting the study. After they collect the data they use analysis 
method 1 to calculate a p-value of 0.04. The researcher asks a statistician to look at the data. The statistician tells 
the researcher the analysis should be done with analysis method 2 and the statistician calculates a p-value of 0.06 
using analysis method 2. Which of the following statements is most correct? 


Answers Teaching Points 


Click Your Answer Below 

The study is statistically si gnif -i- c a- n t beca-use-a-R-aoa-l-ysis- m eth-odHstovided a p-value less than 
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i_i 

The study is statistically sigliifjbfS/Tttbseauap^lai^arisrljffllspia^tilMlCt^dpBMkfted a p-value greater than 
alpha pubmed-reference/id 1447696809? 

The analysis method shoulcfie'pi'cl^d based on what p-value it provides 
The p-value is only valid if the correct analysis method is used 
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